Towards Self-Validating Knowledge-Based Archives

Digital archives are dedicated to the long-term preservation of electronic information and have the mandate to enable sustained access despite a rapidly changing information infrastructure. Current archival approaches build upon standardized data formats and simple metadata mechanisms for collection management, but do not involve high-level conceptual models and knowledge representations. This results in serious limitations, not only for expressing various kinds of information and knowledge about the archived data, but also for creating infrastructure independent, selfvalidating and self-instantiating archives. To overcome these limitations, we first propose a scalable XML-based archival infrastructure, based on standard tools, and subsequently show how this architecture can be extended to a model-based framework, where higher-level knowledge representations become an integral part of the archive and the ingestion/migration processes. This allows us to maximize infrastructure independence by archiving generic, executable specifications of (i) archival constraints (i.e., “model validators”), and (ii) archival transformations that are part of the ingestion process. The proposed architecture facilitates construction of self-validating and self-instantiating knowledge-based archives. We illustrate our overall approach and report on first experiences using a sample collection from a collaboration with the National Archives and Records Administration (NARA). 1 Background and Overview Digital libraries and archives, like their traditional paper-based counterparts, preserve data, information, and knowledge and thus are our “cultural memories” for future generations. While the rapidly evolving information technology provides This research has been sponsored by the National Archives and Records Administration and Advanced Research Projects Agency/ITO, “Intelligent Metacomputing Testbed”, ARPA Order No. D570, issued by ESC/ENS under Contract #F1962896-C-0020 ever-changing new opportunities for storing, managing, and accessing information, the plethora, complexity, and often short life-cycle of storage media, data formats, hardware, and software environments, all contribute to a serious challenge for the long-term preservation of information. Among other findings, the Task Force on Archiving Digital Information concluded that an infrastructure is needed that supports distributed systems of digital archives, and identified data migration as a crucial means for the sustained access to digital information [4]. In a research collaboration with the National Archives and Records Administration (NARA), the San Diego Supercomputer Center (SDSC) developed an information management architecture and prototype for digital archives, based on scalable archival storage systems (HPSS), data handling middleware (SRB/MCAT), and XML-based mediation techniques (MIX) [7, 10, 1].1 A core problem for persistent digital archives is the preservation of data collections in such a way that a faithful representation of their content can be dynamically reinstantiated in the future. To meet this goal, it is not sufficient to merely migrate data at the physically level from obsolete to current media but to create “recoverable” archival representations that are infrastructure independent (or generic) to the largest extent possible. Indeed the challenge is the forward-migration in time of information and knowledge about the archived data, i.e., of the various kinds of meta-information that will allow recreation and interpretation of structure and content of archived data. In this paper, we develop an architecture for infrastructure independent, model-based archival and collection management. Our approach is model-based in the sense that the ingestion process can employ both structural and semantic models of the collection, including a “flattened” relational representation, a “reassembled” semistructured representation, and higher-level “semantic” 1www.clearlake.ibm.com/hpss/, www.npaci. edu/DICE/SRB, and www.npaci.edu/DICE/MIX/